Turner et al. Systematic Reviews 2012, 1:60 
http://www.systematicreviewsjournal.eom/content/1/1/60 



3 SYSTEMATIC 

| REVIEWS 



RESEARCH Open Access 



Does use of the CONSORT Statement impact the 
completeness of reporting of randomised 
controlled trials published in medical journals? 
A Cochrane review 3 

Lucy Turner 1 , Larissa Shamseer 1 , Douglas G Altman 2 , Kenneth F Schulz 3 and David Moher 1,4 * 
Abstract 

Background: The Consolidated Standards of Reporting Trials (CONSORT) Statement is intended to facilitate better 
reporting of randomised clinical trials (RCTs). A systematic review recently published in the Cochrane Library 
assesses whether journal endorsement of CONSORT impacts the completeness of reporting of RCTs; those findings 
are summarised here. 

Methods: Evaluations assessing the completeness of reporting of RCTs based on any of 27 outcomes formulated 
based on the 1996 or 2001 CONSORT checklists were included; two primary comparisons were evaluated. The 27 
outcomes were: the 22 items of the 2001 CONSORT checklist, four sub-items describing blinding and a 'total 
summary score' of aggregate items, as reported. Relative risks (RR) and 99% confidence intervals were calculated to 
determine effect estimates for each outcome across evaluations. 

Results: Fifty-three reports describing 50 evaluations of 16,604 RCTs were assessed for adherence to at least one of 
27 outcomes. Sixty-nine of 81 meta-analyses show relative benefit from CONSORT endorsement on completeness 
of reporting. Between endorsing and non-endorsing journals, 25 outcomes are improved with CONSORT 
endorsement, five of these significantly (a = 0.01). The number of evaluations per meta-analysis was often low with 
substantial heterogeneity; validity was assessed as low or unclear for many evaluations. 

Conclusions: The results of this review suggest that journal endorsement of CONSORT may benefit the 
completeness of reporting of RCTs they publish. No evidence suggests that endorsement hinders the completeness 
of RCT reporting. However, despite relative improvements when CONSORT is endorsed by journals, the 
completeness of reporting of trials remains sub-optimal. Journals are not sending a clear message about 
endorsement to authors submitting manuscripts for publication. As such, fidelity of endorsement as an 
'intervention' has been weak to date. Journals need to take further action regarding their endorsement and 
implementation of CONSORT to facilitate accurate, transparent and complete reporting of trials. 
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Background 

In the mid-1990s, in response to concerns about the 
quality of reporting of randomised controlled trials 
(RCTs), an international group of trialists, statisticians, 
epidemiologists and biomedical editors developed The 
CONsolidated Standards Of Reporting Trials (CON- 
SORT) Statement [1], which has been revised and 
updated twice [2,3], each with a companion explanatory 
document [4,5]. The Statement is an evidence-based 
minimum set of recommendations, consisting of a 
checklist, flow diagram and descriptive text, intended to 
facilitate the complete and transparent reporting of 
RCTs and subsequently aid in their critical appraisal and 
interpretation. Over time, evidence of the impact of 
CONSORT has accumulated. It is one of the most 
widely cited scientific contributions of all time (over 
5,300 citations, not including self-citation). CONSORT 
has also been rated as one of the major milestones in 
health research methods over the last century by the 
Patient-Centered Outcomes Research Institute (PCORI) 
[6]. The most recent version, CONSORT 2010 is among 
the top 1% of article-level content contained in the Pub- 
lic Library of Science (http://www.plos.org/). Its use has 
been endorsed by international organisations of editors 
such as the International Committee of Medical Journal 
Editors (ICMJE), the World Association of Medical Edi- 
tors (WAME), and the Committee On Publication Ethics 
(COPE). Over 600 general and specialty journals cur- 
rently endorse the CONSORT Statement [7]. 

'Endorsement' of CONSORT by a journal is defined as 
any of the following situations, which imply that the 
CONSORT Statement is, at least, in principle incorpo- 
rated into the editorial process of the journal: (a) journal 
editorial statement endorsing the CONSORT Statement: 
either the flow diagram, the checklist or both; (b) re- 
quirement or recommendation in journals Instructions 
to Authors' to follow CONSORT when preparing their 
manuscript; or (c) requirement for authors to submit a 
CONSORT checklist and/or flow diagram with their 
manuscript. At this time, information with regards to 
the regulation and enforcement of use and adherence to 
CONSORT by editors is too sparse to incorporate in this 
review. Complete reporting was assessed by comparing 
the proportion of RCTs adhering to individual CON- 
SORT items, blinding subgroups or by total scores 
across CONSORT checklist items termed 'Total sum 
score'. 

Along with the publication of the 2001 version of 
CONSORT, Moher and colleagues reported an evalu- 
ation of the CONSORT checklist [8]. The authors 
reported that the completeness of reports of RCTs in 
three CONSORT endorsing journals was higher than 
one non-endorsing journal. Since then, additional eva- 
luations have been published which assess the influence 



of CONSORT either directly or indirectly. In 2006 a sys- 
tematic review identified eight evaluations that assessed 
the completeness of reporting in medical journals that 
did or did not formally endorse CONSORT [9]. Despite 
methodological weaknesses of included evaluations, the 
2006 systematic review found that the use of CONSORT 
may improve the quality of reporting of RCTs. 

Objective 

This paper provides a comprehensive summary of the 
91 -page report of the updated systematic review asses- 
sing the influence of CONSORT, published in the 
Cochrane Library [10]. The objective of the update was 
to assess whether journal endorsement of the CON- 
SORT Statement is associated with improved complete- 
ness of reporting of RCTs. 

Methods 

Electronic database searches were performed to update 
the 2006 systematic review for the time period January 
2005 to March 2010, inclusive, searched on 27 May 
2010. The end search date was purposely selected to ex- 
clude the date of publication of the most recent CON- 
SORT Statement (CONSORT 2010) as it was not 
expected it to have been in use for a sufficient period to 
have produced evaluations. 

MEDLINE (via OVID), EMBASE (via OVID), the 
Cochrane Methodology Register and the Cochrane Data- 
base of Systematic Reviews, both using the Wiley inter- 
face, were searched using a comprehensive strategy [10]. 
The following citation indices were also searched using 
the ISI Web of Knowledge interface: The Science Cit- 
ation Index, Social Science Citation Index, and Arts and 
Humanities Citation Index. Electronic records were 
stored and managed online through systematic review 
data management software, DistillerSR® [11]. 

Studies were included in the review if they evaluated 
the completeness of reporting of RCTs, and could be 
included in any of the following comparison groups, by 
assessing the: (1) completeness of reporting assessed in 
RCTs published in CONSORT-endorsing journals com- 
pared to non-endorsing journals; (2) completeness of 
reporting of RCTs published in CONSORT endorsing 
journals before and after endorsement; or (3) complete- 
ness of reporting of RCTs before and after the publica- 
tion of the 1996 or 2001 CONSORT Statement. Studies 
were not excluded based on language of publication or 
validity assessment. The CONSORT checklist is being 
endorsed by non-medical journals, typically in veterinary 
trials [12], such 'non-human' trial evaluations were not 
eligible for this review. 

When the eligibility of studies was either unclear or 
the data reported were insufficient to categorize the 
study into at least one of the reviews' pre-specified 
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comparison groups, corresponding authors were con- 
tacted for clarification and invited to submit unpublished 
data. This enabled judgment about study eligibility and, 
if eligible, provided data which were extracted for meta- 
analyses, when possible. For example, if the citations for 
RCTs included in an evaluation were provided, review 
authors could determine endorsement status of their 
publishing journals at the time of RCT publication by 
checking a journals Instructions to Authors and/or con- 
tacting editors of journals. A journal was considered an 
endorser if it had endorsed CONSORT at least months 
prior to publication of the evaluated RCT. 

Title, abstract and full text screening of potentially 
relevant records was completed by two independent 
reviewers. One reviewer extracted general study charac- 
teristics of included studies, with complete verification 
by a second reviewer. Data on the completeness of 
reporting were extracted by one reviewer; a second re- 
viewer verified accuracy of a 10% random sample. Dis- 
crepancies were resolved by discussion between the two 
reviewers or arbitration with a third member of the re- 
search team, when necessary. No modifications to the 
extracted data were made post-verification. Validity of 
included studies was assessed by one reviewer with 
complete verification by a second reviewer using pre- 
specified criteria [10]. 

Data on 27 outcomes were collected to estimate com- 
pleteness of reporting of RCTs in included evaluations. 
These were: any of the 22 items on the 2001 CONSORT 
checklist, any of four additional sub-items for blinding 
(outcome assessor, intervention, patients and data ana- 
lyst), or 'Total Sum Score' (that is, an aggregate score of 
some or all CONSORT checklist items). While data ex- 
traction was based on items of the 2001 CONSORT 
checklist, where studies evaluated the 1996 checklist, 
analyses were sub-grouped by checklist year since some 
checklist items were known to have undergone substan- 
tial modification in the 1996 to 2001 CONSORT update. 
These items were: 'Title and Abstract', 'Outcomes', 'Sam- 
ple Size', 'Participant Flow' and 'Numbers Analysed' [10]. 

Relative proportions of RCTs adequately reporting any 
of the 27 outcomes were extracted from included stud- 
ies. In addition, if quality of RCTs within included eva- 
luations was assessed using a quality assessment tool 
however measured (for example, Jadad scale [13]), these 
data were extracted. 

Relative risks (RR) and 99% confidence intervals were 
calculated to determine effect estimates for each outcome 
across evaluations. Reported mean scores across some or 
all checklist items, described as a 'Total Sum Score' were 
also pooled where a standardised mean difference (SMD) 
greater than zero indicates more complete reporting 
among RCTs published in CONSORT-endorsing 
journals. 



Results 

The electronic searches retrieved a total of 4,777 
records; two additional reports were identified. In 
addition, eight reports were included in the 2006 sys- 
tematic review. Duplicate records were removed, and 
the titles and abstracts of the remaining 2,896 records 
were screened yielding 632 potentially relevant studies. 
Full text screening identified 53 reports of 50 evaluations 
(Figure 1). Two of the three comparison groups include 
evaluations where more information is known about en- 
dorsement of CONSORT as a more direct intervention. 
Subsequently, we report the findings of these two com- 
parison groups, 'CONSORT endorsing journals com- 
pared to non-endorsing journals' and 'CONSORT 
endorsing journals before and after'. Please refer to the 
Cochrane review for results of the third comparison 
group, cross-sectional samples of RCTs before and after 
the publication of the CONSORT Statement. 

CONSORT-endorsing journals compared to non-endorsing 
journals 

Twenty-nine (of 53) evaluations were eligible for this com- 
parison group. Across the 27 outcomes the number of 
studies per meta-analysis varied (median (IQR1, IQR3), 6 
[5,8]). Adequate reporting of the method of 'Allocation 
Concealment' and the description of flow of participants 
through the trial, 'Participant Flow! had the largest number 
of included studies (#=16), evaluating adequacy of report- 
ing in 2,396 and 2,140 RCTs, respectively. 

Of 27 outcomes evaluated, 25 had effect estimates in- 
dicating a relatively higher proportion of completely 
reported RCTs published in CONSORT-endorsing jour- 
nals compared to non-endorsing journals. Of these, five 
were statistically significant at the 1% level (Figure 2); 
adequate details of method of 'Allocation Concealment' 
RR = 1.81 (1.25, 2.61) (16 evaluations, 2,396 RCTs, l 2 = 75%). 
This suggests that 81% more RCTs published in 
CONSORT-endorsing journals described allocation conceal- 
ment more completely compared to those published in non- 
endorsing journals. 

Other outcomes which resulted in statistically signifi- 
cant effects, favouring the completeness of reporting in 
CONSORT-endorsing journals were providing an ad- 
equate scientific explanation and rationale detailed in 
the 'Introduction' of the trial, RR = 1.07 (1.01, 1.14) (five 
evaluations, 513 RCTs, I 2 = 0%), adequate description of 
how 'Sample Size' was determined RR = 1.61 (1.13, 2.29) 
(11 evaluations, 1,843 RCTs, I 2 = 76%), and adequate de- 
scription of the method used for 'Sequence Generation' 
RR = 1.59 (1.38, 1.84) (14 evaluations, 2,231 RCTs, I 2 = 
24%). Seven included studies evaluating 560 RCTs con- 
tributed to a statistically significant pooled effect in 
favour of CONSORT for 'Total Sum Score': SMD = 0.68 
(0.38, 0.98) (I 2 = 0%). This indicates that when adequate 
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Additional records from other sources (n 2) 
+ 

Evaluations included in 2006 review (n 8) 



Records after duplicates removed 
(n = 2896) 




Full-text articles assessed for 
eligibility 
(n =632) 



Potentially eligible evaluations 
(n = 64) 



r 



Records excluded 
(n = 2264) 



□ 



Full-text articles excluded (n 569) 
Reasons for exclusion: 

Did not include a comparison group (n = >6>t 
Did not report any o f the 22 items of the 
CONSOR T checklist or a total score (n 52H) 
Did not assess the influence of CONSORT on the 
quality of RCTs (n 516) 
Articles yvere unavailable (n=lO) 
Studies considered use of the CONSORT checklist 
[for non-human RCTs (n=H) 



Additional studies excluded (n 11) 

Reason for exclusion: 

data not available from authors 



Evaluations included in qualitative 
synthesis (n = 53) 



7 



Evaluations included in quantitative 
synthesis ( meta-analyses | 
(n ~ 4 ( ) reports i 



Figure 1 PRISMA flow diagram. 



reporting was summarised across all CONSORT items, 
RCTs published in CONSORT-endorsing journals were 
more completely reported than RCTs published in non- 
endorsing journals. 

Precise details of Interventions', CONSORT checklist 
item four, were equally reported in endorsing and non- 
endorsing journals, RR=1.0 (0.95, 1.05) (six evaluations, 
638 RCTs, I 2 = 0%), and eligibility criteria for trial 'Parti- 
cipants' was reported in six evaluations assessing a total 
of 683 RCTs, and resulted in an effect estimate less than 
1.0, RR= 0.95 (0.56, 1.62). This suggests that the relative 
completeness of reporting for this item is slightly less in 
CONSORT-endorsing journals compared to non- 
endorsing journals. 

Eight studies were not strictly compliant with the def- 
inition of a CONSORT-endorsing journal used in this 
review. Sensitivity analyses showed that only one out- 
come (of 27), although only minimally different, differed 



when evaluations that did not directly meet our defin- 
ition of endorsement were excluded. Completeness of 
reporting of the scientific rational and background in 
the Introduction, was adjusted from RR = 1.07 (1.01, 
1.14) to 1.05 (0.87, 1.27). 

CONSORT-endorsing journals before and after CONSORT 
endorsement 

Eleven (of 53) evaluations assessed journals that endorse 
the CONSORT Statement and presented RCT complete- 
ness of reporting of at least one CONSORT item before 
and after the journals date of endorsement of CON- 
SORT. The number of RCTs assessed per outcome had 
a median (IQR) of 532 (512, 919). The number of 
reported CONSORT checklist items varied over evalua- 
tions, with a median (IQR) of 3 (2, 5). Adequate report- 
ing of the method of 'Sequence generation' and the flow 
of participants through the trial, 'Participant Flow', were 
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#of 



#of 



CONSORT Checklist Item Evaluations 


RCTs 


RR 


99% CI 


Title and /Abstract 


7 


1,233 


1.13 


(0.96,1.33) 


Introduction 


5 


513 


1.07 


(1 .01,1.14) 


Participants 


6 


683 


0.95 


(0.56,1.62) 


Interventions 


6 


638 


1.00 


(0.95,1.05) 


Objectives 


5 


540 


1.01 


(0.96,1.06) 


Outcomes 


8 


1,302 


1.17 


(0.95,1.44) 


Sample Size 


11 


1,843 


1.61 


(1 .13,2.29) 


Seq uen ce G en erati on 


14 


2,231 


1.59 


(1 .38,1.84) 


Al 1 o cation C o nceal m ent 


16 


2,396 


1.81 


(1 .25,2.62) 


Implementation 


5 


498 


1.47 


(0.65,3.32) 


Bl i n d ing o f P ati cipants 


5 


711 


1.39 


(0.87,2.22) 


Bl i n d ing o f 1 nter vention 


5 


710 


1.25 


(0.74,2.12) 


Blinding of Outcome Assessor 


5 


719 


1.72 


(0.69,4.30) 


Blinding of Data Analyst 


3 


497 


3.56 


(0.40,31 .8) 


Blinding Any description 


8 


1,851 


1.23 


(0.93,1.62) 


Statistical Methods 


9 


894 


1.03 


(0.90,1.18) 


Participant Flow 


16 


2,461 


1.16 


(0.94,1.44) 


Recruitment 


6 


959 


1.03 


(0.75,1.41) 


Baseline Data 


5 


529 


1.07 


(0.94,1.22) 


Numbers Analysed 


13 


2,145 


1.23 


(0.98,1.55) 


Outcomes and Estimation 


6 


617 


1.00 


(0.95,1.05 


An ciliary Analyses 


4 


378 


1.31 


(0.48,3.58) 


Adverse Events 


8 


911 


1.14 


(0.86,1.52) 


Interpretation 


5 


540 


1.01 


(0.96,1.06) 


Generalisability 


5 


540 


1.22 


(0.88,1.70) 


Overall Evidence 


4 


317 


1.03 


(0.91,1.17) 



Favours 
Non-Endorsement 



Favours 
Endorsement 



0.5 



1 



Pooled Risk Ratios and 99% CI 

Figure 2 Pooled relative risks across assessed 2001 CONSORT checklist items with 99% confidence intervals for primary comparison, 
adherence of RCTs published in CONSORT-endorsing journals compared to RCTs published in CONSORT non-endorsing journals. 



both reported in eight evaluations. For 15 of 27 out- 
comes data were reported in fewer than five evaluations. 
The results across all outcomes in this comparison are 
presented (Figure 3). 

Discussion 

Serious systemic problems exist in how research is 
reported. This is a wasteful use of already limited 



resources that fund health research [14]. This issue might 
reflect several factors including inadequate conduct, slop- 
piness in reporting, and quite possibly an inability of 
researchers to clearly and transparently inform readers 
about the methods and findings of their research [15]. The 
problem is endemic known to affect many, if not all, areas 
of healthcare research [16-18]. Systematic reviews evaluat- 
ing the effectiveness of health interventions typically rely 



CO II SO RT Checklist Item 


#of 


#of 


RR 


99% CI 




Evaluations 


RCTs 






Title and Abstract 


3 


532 


1.41 


(0.63,3.16) 


Introduction 


2 


457 


1.04 


(1 .00,1.08) 


Participants 


4 


622 


0.98 


(0.88,1.09) 


Interventions 


4 


630 


1.02 


(0.97,1.07) 


Objectives 


2 


517 


1.04 


(0.91,1.18) 


Outcomes 


5 


716 


1.43 


(0.85,2.40) 


Sample Size 


6 


983 


1.30 


(0.71,2.37) 


Seq uen ce G en erati on 


8 


1,085 


1.46 


(0.88,2.44) 


fi-l 1 o cation C o nceal m ent 


6 


855 


1.23 


(0.55,2.75) 


Implementation 


2 


517 


1.94 


(0.15,24.7) 


Blinding of Participants 


1 


75 


0.77 


(0.45,1.31) 


Bl i n d ing o f 1 nter vention 


1 


75 


0.26 


(0.09,0.74) 


Bl i n d ing o f 0 utcom e Assessor 


1 


75 


0.66 


(0.34,1.30) 


Blinding of Data Analyst 


1 


75 


0.27 


(0.02,3.71) 


Bl i n d ing An y d escription 


4 


926 


0.96 


(0.61,1.51) 


Statistical Methods 


5 


1,111 


0.86 


(0.62,1.19) 


Participant Flow 


S 


992 


1.33 


(0.95,1.87) 


Recruitment 


3 


326 


1.77 


(0.48,6.49) 


Baseline Data 


2 


529 


1.42 


(1 .24,1.62) 


Numbers Analysed 


6 


1,005 


1.72 


(1 .18,2.50) 


Outcomes and Estimation 


3 


532 


1.35 


(0.73,2.50) 


j&n ciliary Analyses 


1 


442 


3.46 


(2.47,4.84) 


Adverse Events 


3 


507 


1.39 


(1 .12,1.73) 


Interpretation 


1 


442 


1.01 


(0.98,1.04) 


Generalisability 


1 


442 


1.77 


(2.12,1.48) 


Overall Evidence 


2 


517 


1.31 


(0.99,1.73) 



Favours 
Non-Endorsement 



0.5 



Favours 
Endorsement 



1 



Pooled Risk ratios and 99% CI 

Figure 3 Pooled relative risks across assessed 2001 CONSORT checklist items with 99% confidence intervals comparing the adherence 
of RCTs published in CONSORT-endorsing journals before and after endorsement. 
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on reports of RCTs as their primary source of evidence 
[19]. However, the inadequate reporting of essential ele- 
ments in these reports hinders the systematic review 
process, often leaving reviewers unable to make definitive 
conclusions [20]. 

This review suggests that RCTs which report their 
findings using the CONSORT statement as guidance 
may produce more complete RCT reports. While this 
review provides some important insight into the impact 
of CONSORT on completeness of reporting it is not 
without limitations. Although the search was completed 
more than 2 years ago we do not believe that any new 
evaluations would alter the present results as the 
included number of evaluations is large and includes 
data from over 16,600 trials. The majority of CONSORT 
evaluations are small and very large studies with dra- 
matic effect estimates would need to be included to 
modify the current results. Similarly, newer evaluations 
may evaluate the CONSORT 2010 checklist which are 
ineligible for this systematic review. None of the 
included evaluations utilized an experimental design 
(that is, randomised trial) and as such are subject to the 
influence of confounding, in particular, potential im- 
provement in completeness of reporting over time as 
well as discrepant editorial policies between journals. 
The inclusion criteria for this review were broad and 
variability in methodology, field of interest and validity 
between evaluations was considerable. The use of 99% 
confidence intervals, intended to estimate conservative 
effects given the questionable validity and heterogeneity 
of included studies, aims to offer readers with more con- 
fidence in the findings of this review, especially those of 
statistical significance. 

There are inconsistencies in how journals implement 
reporting guidelines when reports of RCTs are submit- 
ted for publication [21,22]. Since editorial procedures 
are not consistently available online, this review was un- 
able to assess journal implementation of CONSORT en- 
dorsement (that is, verification by the journals' editorial 
team of author adherence to CONSORT). Anecdotally, 
there is variability between how verification for CON- 
SORT adherence by authors factors into the editorial 
decision-making process (if at all) between journals. 
Editors may want to use these results develop explicit 
statements about their journals' endorsement of CON- 
SORT, and other reporting guidelines, in their Instruc- 
tions to Authors', and optimally to recommend 
submission of the CONSORT checklist and flow dia- 
gram at the time of manuscript submission [23]. It is 
likely that active endorsement policies by journals will 
lead to more complete, clear and transparent publica- 
tions, ultimately increasing the usability of research 
reports to make decisions about healthcare treatment 
and services. 



Conclusions 

The results of this review suggest that journal endorse- 
ment of CONSORT may benefit the completeness of 
reporting of RCTs they publish. No evidence suggests 
that endorsement hinders the completeness of RCT 
reporting. However, despite relative improvements when 
CONSORT is endorsed by journals, the completeness of 
reporting of trials remains sub-optimal. Journals are not 
sending a clear message about endorsement to authors 
submitting manuscripts for publication. As such, fidelity 
of endorsement as an 'intervention' has been weak to 
date. Journals need to take further action regarding their 
endorsement and implementation of CONSORT to fa- 
cilitate accurate, transparent and complete reporting of 
trials. 

All CONSORT guidance documents including the 
2010 checklist, flow diagram, extension statements and 
other resources are freely available from the CONSORT 
website [24] and in numerous open-access publications. 
Their consistent use and adherence by authors and jour- 
nals has the potential to revolutionize the reporting and 
subsequent usefulness of RCT reports. 
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